(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 

International Bureau 



lllllllllllllllllllllllllllllllllllllllllllllllli 



(43) International Publication Date 
21 November 2002 (21.11.2002) 



PCT 



(10) International Publication Number 

WO 02/093877 Al 



(51) International Patent Classification 7 : 

3/00, 3/42, 11/10, H04B 1/38 



H04M 1/00. 



(74) Agent: HOEL, John; Morgan & Finnegan, LLP, 345 Park 
Avenue, New York, NY 10154 (US). 



(21) International Application Number: PCT/IB02/01550 

(22) International Filing Date: 7 May 2002 (07.05.2002) 

(25) Filing Language: English 

(26) Publication Language: English 

(30) Priority Data: 

09/854,628 15 May 2001 (15.05.2001) US 

(71) Applicants: NOKIA CORPORATION [FI/FI] ; Keilalah 
dentie 4, FTN-02150 Espoo (FT). NOKIA INC. [US/US]; 
6000 Connection Drive, Irving, TX 75039 (US). 

i (72) Inventors: NYKANEN, Petri; Lehdokkikatu 3, 

j FIN-37120 Nokia (Fl). PALON1EM1, Jari; Yliki- 

j imingintie 169, FIN-90900 Kiiminki (FI). KANGAS, 

j Petri; Polkkytie 7A3, FIN-90800 Oulu (FI). 



(81) Designated States (national): AE, AG, AL, AM, AT, AU, 
AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CU, 
CZ, DE, DK, DM, DZ, EC, EE, ES, FI, GB, GD, GE, GH, 
GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, 
LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, 
MX, MZ, NO, NZ, OM, PH, PL, PT, RO, RU, SD, SE, SG, 
SI, SK, SL, TJ, TM, TN, TR, TT, TZ, UA, UG, UZ, VN, 
YU, ZA, ZM, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZM, ZW), 
Eurasian patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), 
European patent (AT, BE, Cll, CY, DE, DK. liS, Fl, FK, 
GB, GR, IE, IT, LU, MC, NL, PT, SE, TR), OAPI patent 
(BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, ML, MR, 
NE, SN, TD, TG). 

Published: 

— with international search report 



j (54) Title: CONTEX'I SENSITIVE WEB SERVICES 



oo 
m 

On 

o 



CONTEXT SENSITIVE WEB SERVICES 

SESSION TYPE: 

[A] UPDATE PRIVACY FEATURES OR 

[B] RUN AN APPLICATION 



UPDATE PRIVACY FEATURES: 

[ 1 ] UPDATE YOUR PRIVACY PROFILE 
[2) UPDATE YOUR PERSONAL DATA 
[3| AUTHENTICATE A PROGRAM 



POSITIONING SENSOR 122 



TOUCH SENSOR 124 | AUDIO SENSOR 125 



| AMBIENT TEMPERATURE SENSORTX 



| THREE-AXIS ACCELERATION SENSOR iT 



APPLICATION PGMS 106 



WAP CLIENT PROGRAM 108 




- (57) Abstract: The context sensitive web services method enables a mobile phone or wireless device (100) to use conte 
J techniques to sense the user' s environment and in response, to provide useful information to the user that is appropriate 
perceived environment. The method includes the steps of receiving sensor signals (122-134) characterizing a current 
J of the wireless device (100); processing the sensor signals with a context inference engine (142); outputting a current 
► from the processing by the context inference engine (142); and providing useful information to the user in response 
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CONTEXT SENSITIVE WEB SERVICES 

This application claims priority to U.S. Application Serial No. 09/857,628, filed 
5 May 15, 2001, entitled, "Context Sensitive Web Services, which is incorporated herein by 
reference. 

BACKGROUND OF THE INVENTION 

10 Field of the Invention: 

The invention disclosed broadly relates to methods for providing Internet services 
and more particularly relates to improvements in mobile device accessing of Internet 
services. 

15 

Background Art: 

Mobile phones and wireless personal digital assistants (PDAs) are able to access 
the Internet using the Wireless Application Protocol (WAP). WAP-enabled wireless 

20 devices can now access Internet applications such as headline news, exchange rates, 

sports results, stock quotes, weather forecasts, multilingual phrase dictionaries, personal 
online calendars, online travel and banking services, or download distinctive ringing 
tones. Broadband wireless networks make it possible for WAP-enabled wireless devices 
to exchange multimedia messages that combine conventional text with much richer 

25 content types, such as photographs, images, voice clips, and video clips. WAP-enabled 
wireless devices can be used to pay bills online using the wireless device as a virtual 
wallet. WAP-enabled wireless devices can deliver useful and informative advertising and 
transaction services from online merchants. WAP-enabled wireless devices now also 
provide entertainment services, such as interactive adventure games, quizzes, and chess 

30 tournaments. 

What is needed is the ability of a mobile phone or wireless PDA to use context 
inference techniques to sense the mobile user's environment and in response, to provide 
useful information to the user that is appropriate to the user's perceived environment. It 
would be even more useful to offload some of the computationally intensive computing 
35 necessary in context inference techniques, from the mobile user's wireless device to a 
server and to web sites on the Internet. It would be beneficial to maintain a personal 
1 
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profile of the mobile user's personal preferences in an online server or web site. It would 
be important to provide the mobile user with the ability to control any access to the user's 
profile by the online server or web site. 

5 

SUMMARY OF THE INVENTION: 

The context sensitive web services invention enables a mobile phone or wireless 
PDA to use context inference techniques to sense the user's environment and in response, 
10 to provide useful information to the user that is appropriate to the user's perceived 
environment. 

One aspect of the invention is a method to enable a wireless device to provide 
useful information to its user that is appropriate to the device's current environment. The 

15 method includes the steps of receiving sensor signals characterizing a current environment 
of the wireless device; processing the sensor signals with a context inference engine; 
outputting a current context result from the processing by context inference engine; and 
providing useful information to the user in response to the current context result. The 
processing of the sensor signals with a context inference engine is embodied as 

20 programmed instructions executed within the user's wireless device. In another aspect of 
the invention, the processing of the sensor signals with a context inference engine is 
embodied as programmed instructions executed within a separate network server in 
response to signals from the user's wireless device. The server can access files from a web 
server, for selective forwarding to the user's wireless device. A personal profile of the user 

25 can be maintained by the server. 

A further aspect of the invention provides user control of access by application 
programs to the user's private data. This can also include providing user control of access 
by application programs to the user's private data in the server. Still further, this can also 
include providing user control of access by application programs in a web server, to the 

30 user's private data. 

Another aspect of the invention is providing the current context result to an 
application program in response to the user control and receiving the useful information 
from the application program. The invention enables the user to grant access permission 
to the application program to access the current context result. This can be performed in 
2 
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the user's wireless device or in the network server. The network server can carry out the 
control of access by application programs in web servers, in response to a user privacy 
profile received from the user's wireless device. 

5 

DESCRIPTION OF THE FIGURES: 

Figure 1 is a network diagram of the invention, showing an example relationship 
between the user's Wireless Application Protocol (WAP)-enabled portable wireless 
10 device, the WAP protocol gateway to the Internet, the network server, the Universal 
Description, Discovery and Integration (UDDI) registry, and a plurality of web sites. 

Figure 1A shows the user's wireless device with the UPDATE PRIVACY 
FEATURES: sub-menu of the Context Sensitive Services menu, enabling the user to 
UPDATE YOUR PRIVACY PROFILE or UPDATE YOUR PERSONAL DATA. 
15 Figure IB shows the user's wireless device with the UPDATE PRIVACY 

FEATURES: sub-menu of the Context Sensitive Services menu, enabling the user to 
AUTHENTICATE A PROGRAM and REGISTER A PROGRAM. 

Figures 1C and ID show the user's wireless device with the RUN AN 
APPLICATION sub-menu of the Context Sensitive Services menu, enabling the user to 
20 RUN AN APPLICATION. 

Figure 2 is a functional block diagram of the wireless device 100, showing its 
various components and programs. 

Figure 2A is a functional block diagram of the wireless device 100, the server 140, 
and the web server 160, and their interaction when exchanging a metadata vector 138 and 
25 privacy control data 150. 

Figure 3 is a network process flow diagram of the interaction of the wireless 
device 100, network server 140, and web server 160 when carrying out the determination 
of the current context of the wireless device 100. 

Figure 4 is a functional block diagram of the network server 140, showing the 
30 memory storing the application services software programs needed to perform the 
operations of the invention. 

DISCUSSION OF THE PREFERRED EMBODIMENT: 



3 
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The context sensitive web services invention enables a mobile phone or wireless 
PDA to use context inference techniques to sense the user's environment and in response, 
to provide useful information to the user that is appropriate to the user's perceived 
environment. The invention offloads some of the computationally intensive computing 
5 necessary in context inference techniques, from the mobile user's wireless device to a 
server and to web sites on the Internet. The context sensitive web services invention 
maintains a personal profile of the mobile user's personal preferences in an online server 
or web site. The mobile user is provided with the ability to control access by application 
programs in the wireless device, to the user's private data. The context sensitive web 

10 services invention provide the mobile user with the ability to control any access to the 
user's profile by the online server or web site. 

The mobile user's wireless device is equipped with a context inference engine for 
providing and awareness of the mobile user's context to application programs, including 
third party applications. Since the processing power and storage capacity is limited in 

15 typical wireless devices, the computational load and storage requirements of the context 
inference engine are distributed to a context inference server capable of processing the 
context data. The invention enables the mobile user to control which application 
programs in the wireless device are granted access to the user's private context 
information. A privacy control block in the wireless device grants or revokes access by 

20 application programs to the private context information, based on the mobile user's 

preferences stored in a privacy profile. The same privacy control and privacy profile is 
extended to the context inference server, thereby enabling the extension of the user's 
privacy control to any web server connected to the context inference server. The 
invention thus enables building an infrastructure for context sensitive applications and 

25 services within the wireless device and the server, while providing to the mobile user 
control over the privacy user's context information. 

The invention is applied to wireless telephones and wireless personal digital 
assistants (PDAs) implementing the Wireless Application Protocol (WAP) standard. 
Figure 1 is a network diagram of an embodiment of the invention, showing an example 

30 relationship between the user's Wireless Application Protocol (WAP)-enabled portable 
wireless device 100, a WAP protocol gateway 120, and the server 140. The user's WAP- 
enabled portable wireless device 100 can be a wireless mobile phone, pager, two-way 
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radio, smartphone, personal communicator, or the like. The user's WAP-enabled portable 
wireless device 100 accesses a small file called a deck which is composed of several 
smaller pages called cards which are small enough to fit into the display area of the 
device's microbrowser 102. The small size of the microbrowser 102 and the small file 
5 sizes accommodate the low memory constraints of the portable wireless device 100 and 
the low-bandwidth constraints of a wireless network 1 16. The cards are written in the 
Wireless Markup Language (WML) which is specifically devised for small screens and 
one-hand navigation without a keyboard. The WML language is scaleable from two-line 
text displays on the microbrowser 102 of a cellular telephone, up through large LCD 
10 screens found on smart phones and personal communicators. The cards written in the 
WML language can include programs written in WMLScript, which is similar to 
JavaScript, but makes minimal demands on memory and CPU power of the device 100 
because it does not contain many of the unnecessary functions found in other scripting 
languages. 

15 The Nokia WAP Client Version 2.0 is a software product containing the 

components necessary to implement the WAP client 108 on the wireless device 100. 
These components include a Wireless Markup Language (WML) Browser, WMLScript 
engine, Push Subsystem, and Wireless Protocol Stack. The Nokia WAP Client is a 
source-code product that can port and integrate into wireless devices such as mobile 

20 phones and wireless PDAs. Application programs 106 stored in the wireless device 100 
interact with the WAP Client 108 to implement a variety of communications applications. 
Details of the Nokia WAP Client Version 2.0 can be found in the online paper: Nokia 
WAP Client Version 2.0, Product Overview , Nokia Internet Communications, 2000, 
www.nokia.com/corporate/wap. 

25 The WAP Client 108 includes the Wireless Public Key infrastructure (PKI) 

feature, providing the infrastructure and the procedures required for authentication and 
digital signatures for servers and mobile clients. Wireless PKI is a certificate-based 
system that utilizes public/private key pairs associated with each party involved in a 
mobile transaction. Wireless Identity Module (WIM) is a security token feature of the 

30 WAP Client 108, which includes security features, such as the public and private keys and 
service certificates, needed for user authentication and digital signatures. Additionally, it 
has the ability to perform cryptographic operations to encrypt and decrypt messages. 
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The wireless device 100 of Figure 1 also has a plurality of sensors for sensing the 
mobile user's ambient conditions. The sensors shown include POSITIONING SENSOR 
122, TOUCH SENSOR 124, AUDIO SENSOR 125, COMPASS SENSOR 126, 
AMBIENT LIGHT SENSOR 128, AMBIENT TEMPERATURE SENSOR 132, and 
5 THREE-AXIS ACCELERATION SENSOR 134. The audio sensor 125 can be a 
microphone, for example, which can detect speech or environmental sounds. The 
positioning sensor can be, for example, a GPS receiver integrated in the device. The 
positioning sensor can also be, for example, a radio beacon triangulation sensor that 
determines the location of the wireless device by means of a network of radio beacons, 
10 base stations, or access points, as is described for example, in Nokia European patent EP 
0 767 594 A2, entitled "Mobile Station Positioning System". These sensors provide 
inputs which are sampled by the wireless device 100 to infer a current context, as will be 
described below. 

The WAP protocol gateway 120 links the Internet 130 and the wireless network 

15 1 16. The WAP protocol gateway 120 includes the Wireless Public Key infrastructure 

(PKI) feature to help provide a secure Internet connection to the wireless device 100. The 
WAP protocol gateway 120 enables the WAP-enabled wireless device 100 to access 
Internet applications such as headline news, exchange rates, sports results, stock quotes, 
online travel and banking services, or to download distinctive ringing tones. 

20 The user's WAP-enabled portable wireless device 100 communicates with the 

radio tower 1 14 and can exchange messages for distances up to several kilometers. The 
types of wireless networks 1 16 supported by the WAP standard include Cellular Digital 
Packet Data (CDPD), Code-Division Multiple Access (CDMA), Global System for 
Mobile Communications (GSM), Time Division Multiple Access (TDMA), GPRS, 3G- 

25 Broadband, and the like. 

The overall process of communication between the user's WAP-enabled wireless 
device (the client) 100, through the WAP protocol gateway 120, to the server 140 
resembles the way Web pages are served on the Internet using the HyperText Transfer 
Protocol (HTTP) or World Wide Web protocol: 

30 [1] The user presses a phone key on the user's device 100 related to the Uniform 

Resource Locator (URL) of the server 140. 
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[2] The user's device 100 sends the URL, via the radio tower 114 and the wireless 
network 116, to the gateway 120 using WAP protocols. 

[3] The gateway 120 translates the WAP request into an HTTP request and sends 
it over the Internet 130 to the server 140, via Transmission Control Protocol/ Internet 
5 Protocol (TCP/IP) interfaces. 

[4] The server 140 handles the request just like any other HTTP request received 
over the Internet. The server 140 either returns a WML deck or a HyperText Markup 
Language (HTML) page back to the gateway 120 using standard server programs written, 
for example in Common Gateway Interface (CGI) programs, Java servlets, or the like. 
10 [5] The gateway 120 receives the response from the server 140 on behalf of the 

user's device 100. If the response is an HTML page, it gets transcoded into WML if 
necessary. Then the WML and WMLScript coding is encoded into a byte code that is 
then sent to the user's device 100. 

[6] The user's device 100 receives the response in the WML byte code and 
15 displays the first card in the deck on the microbrowser 102 to the user. 

In Figure 1, the protocol gateway 120 includes a WAP protocol stack organized 
into five different layers. An application layer is the wireless application environment, 
which executes portable applications and services. A session layer is the wireless session 
protocol, which supplies methods for the organized exchange of content between 
20 client/server applications. A transaction layer is the wireless transaction protocol, which 
provides methods for performing reliable transactions. A security layer is the wireless 
transport layer security, which provides authentication, privacy, and secure connections 
between applications. The transport layer is the wireless datagram protocol, which 
shelters the upper layers from the unique requirements of the diverse wireless network 
25 protocols, such as CDPD, CDMA, GSM, etc. Additional information about the WAP 
standard and the WAP protocol stack can be found in the book by Charles Arehart, et al. 
entitled, "Professional WAP", published by Wrox Press Ltd., 2000 (ISBN 1-861004-04- 
1). 

In Figure 1, the user's portable wireless device 100 includes the microbrowser 102 
30 displays the Context Sensitive Services menu, to enable the user to navigate through the 
cards being displayed and to select options that are programmed by the application 
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programs 106. The user's device 100 also includes the WAP client program 108 which 
has been previously discussed. 

The Context Sensitive Services menu displayed by the microbrowser 102 in 
Figure 1 is rendered by the WAP client program 108 under the control of the application 
5 programs 106, which are shown in Figures 2 and 2A. The user can select the session type 
with Context Sensitive Services menu, either [A] UPDATE PRIVACY FEATURES or 
[B] RUN AN APPLICATION. If the UPDATE PRIVACY FEATURES session type is 
selected by the user, the Context Sensitive Services menu of Figure 1 then presents to the 
user the UPDATE PRIVACY FEATURES sub-menu from which the user can select the 
10 following options: 

[A] UPDATE PRIVACY FEATURES: 

[1] UPDATE YOUR PRIVACY PROFILE 
[2] UPDATE YOUR PERSONAL DATA 
1 5 [3] AUTHENTICATE A PROGRAM 

Option [1] of UPDATE YOUR PRIVACY PROFILE, leads to a second sub-menu 
shown in Figure 1A, which has the following options: 

20 [1] UPDATE YOUR PRIVACY PROFILE 

[a] Add a local program to permissions list 

[b] Remove a local program from list 

[c] Add a server program to permissions list 

[d] Remove a server program from list 

25 [e] Add a network program to permissions list 

[f ] Remove a network program from list. 

Option [2] of UPDATE YOUR PERSONAL DATA, leads to a another sub-menu 
shown in Figure 1A, which has the following options: 

30 

[2] UPDATE YOUR PERSONAL DATA 

[a] Update server database 

[b] Update network database. 

35 Option [3] of AUTHENTICATE A PROGRAM, leads to a another sub-menu 

shown in Figure IB, which has the following options: 



40 



[3] AUTHENTICATE A PROGRAM 

[a] Request program's public key certificate 

[b] Verify certificate signatures 
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[c] Verify validity time 

[d] Verify revocation status 

[e] Check if certificate authority on trust list 
[f ] Flag program as authenticated. 

5 

The AUTHENTICATE A PROGRAM option calls the privacy control 150 of the 
wireless device 100 in Figure 2. If an application program A, B, X, or Y has been verified 
for its acceptability by a trusted authority, then the trusted authority will have issued a 
digital certificate on a message authentication code (MAC) it has computed for the 

10 application program, which can be checked by the privacy control 150. As long as the 
privacy control 150 trusts the trusted authority issuing the digital certificate, 
authentication of the application program is straight forward. 

Once the mobile user has verified the program's digital certificate and is satisfied 
that the application program will not subvert the integrity or security of the user's private 

15 data, the user can register the program. Registration is the granting by the user of access 
permission to the program, to access the current context of the user's wireless device 
and/or to access other portions of the user's private data. There are several levels of 
permission that can be granted by the user in two categories, [a] when can the accesses 
take place and [b] what data can be accessed. 

20 Option [4] of REGISTER A PROGRAM, leads to a another sub-menu shown in 

Figure IB, which has the following options: 

[4] REGISTER A PROGRAM 

[a] When can the accesses take place 
25 [b] What data can be accessed 

For the first category of [a] when can the accesses take place, the highest level of 
permission in this category is that access can occur anytime and without notice. The 
lowest level of permission in this category is that access can only occur at specified times 

30 or under specified conditions, and only after notice to the user and specific authorization 
by the user. For the second category of [b] what data can be accessed, the highest level of 
permission in this category is to access unlimited datasets in the user's private data, 
including current context information, personal data entered by the user, the user's Internet 
usage history data, the user's Internet cookie data, and the user's application program 

35 usage data. The lowest level of permission in this category is that access of any data can 
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only occur after notice to the user and specific authorization by the user. The user can 
configure any levels of permission in between the highest and lowest and make that the 
basis for the registration. The user can include the terms of registration in a digital 
certificate signed by the user and appended to the application program. This registration 
5 certificate can be presented by the program to the privacy control 150 prior to a proposed 
access event, the privacy control 150 to automatically verify the registration status of the 
program. The registration certificate can be constructed as follows. 

The privacy control 150 can compute a message authentication code (MAC) and 
10 its own digital signature and append it as a certificate to an acceptable application 

program A, B, X, or Y. The privacy control 150 can include the terms of registration in 
the digital certificate. Then when the program requests access to the user's private data, 
the privacy control 150 can automatically check the MAC and its own digital signature to 
verify that the program has not been changed and the privacy control 150 can also 
15 automatically verify the registration status of the program. This is achieved by the privacy 
control 150 computing a hash value for the entire application program A, B, X, or Y (or 
some portion of it) and the terms of registration, and then forming a message 
authentication code (MAC) from the hash value. The privacy control 150 then uses its 
PKI private key to digitally sign the message authentication code (MAC). The terms of 
20 the registration, the MAC and the privacy control's digital signature are appended to the 
application program A, B, X, or Y as a registration certificate. 

Then, whenever the application program A, B, X, or Y requests access to the 
user's context data or private data, the privacy control 150 will require the application 
25 program to present the registration certificate so that the privacy control 150 can check 
that the presented MAC compares with a computed MAC and that the presented digital 
signature is genuine. The privacy control 150 can then automatically grant access 
permission to the application program, in accordance with the terms of the registration. 

30 Methods to generate and evaluate message authentication codes to insure the 

integrity of data are described in the book by Stephen Thomas entitled "SSL and TLS", 
published by John Wiley and Sons, 2000. Two example algorithms for message 
authentication are RSA's Message Digest (MD5) and the Secure Hash Algorithm (SHA), 

10 
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both of which are described in the book by Stephen Thomas. Another reference that goes 
into greater detail in its discussion of data integrity methods is the book by Bruce 
Schneier entitled "Applied Cryptography - 2nd Edition" published by John Wiley and 
Sons, 1996. Methods to generate and evaluate digital signatures to insure the source of the 
digital program are described in the book by Richard E. Smith entitled "Internet 
Cryptography", published by Addison Wesley, 1997. 

What has been described here for the privacy control 150 in the wireless device 
100, is equally applicable to the privacy control 164 in the network server 140 of Figure 
2A. The privacy control 164 in the network server 140 can compute the message 
authentication code (MAC) and its own digital signature and append it, with the terms of 
the registration, as a registration certificate to an acceptable application program in the 
web server 160. Privacy control 164 has a cached copy 144 of the Privacy Profile 152 of 
the wireless device 100. This enables automatically processing the privacy check in the 
network Server 140 for access requests from web server 160. When the application 
program in the web server 160 requests access to the user's private data in the network 
server 140 or in the wireless device 100, the privacy control 164 in the network server 140 
will require the application program in the web server 160 to present the registration 
certificate so that it can check the MAC and its own digital signature to verify that the 
application program has not been changed. The privacy control 164 can then 
automatically grant access permission to the application program in the web server 160, in 
accordance with the terms of the registration. 

If the [B] RUN AN APPLICATION session type is selected by the user, the 
Context Sensitive Services menu of Figure 1C then presents to the user the RUN AN 
APPLICATION sub-menu from which the user can select the following options: 

[1] MESSAGING 
[a] exchange multimedia messages 

[2] MOBILE COMMERCE 

[a] personal online calendar 

[b] exchange rates 

[c] banking services 

[d] pay bills online using virtual wallet 

[e] useful and informative advertising 

11 
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[f] online merchant transaction services 

The Context Sensitive Services menu of Figure ID presents to the user the RUN 
AN APPLICATION sub-menu from which the user can select the following options: 

[3] ENTERTAINMENT 

[a] headline news 

[b] sports results 

[c] stock quotes 

[d] weather forecasts 

[e] multilingual phrase dictionaries 

[f] online travel 

[g] download distinctive ringing tones 

[h] interactive games 

The RUN AN APPLICATION option calls one of the application programs A, B, 
X, or Y of the wireless device 100 in Figure 2. 

Figure 2 is a functional block diagram of the wireless device 100, showing its 
various components and programs. The wireless device 100 has context sensitive 
applications A, B, X, and Y, either downloaded, or in firmware. The wireless device 100 
does not need to utilize external functionality in the network for the initial sampling and 
digitization of the sensor inputs. The sampled and digitized values of the sensor inputs 
are POSITIONING METADATA 122', TOUCH METADATA 124', AUDIO 
METADATA 125', COMPASS METADATA 126*, AMBIENT LIGHT METADATA 
128', AMBIENT TEMPERATURE METADATA 132', and THREE-AXIS 
ACCELERATION METADATA 134'. The sampled and digitized values of the sensor 
inputs are loaded into a metadata vector 138. 

Figure 2 shows the memory 202 of the wireless device 100, connected by the bus 
204 to the keypad 104, the radio 206, the sensor interface 208, the central processor 210, 
and the display 212. The memory 202 stores programs which are sequences of executable 
instructions which, when executed by the processor 210, carry out the methods of the 
invention. The memory 202 stores the WAP client program 108, the context inference 
engine 136, the privacy control 150, the privacy profile 152, the context aware API 154, 
the motion/gesture API 156, the location API 158, and other APIs 162. The context 
inference engine 136 processes the metadata vector 138 to produce the current context. 

12 
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Application programs 106 stored in the memory 202 include the application programs A 
and B which are part of the software system SSI, and the application programs X and Y 
which are contained in the execution environment "Exec. Env." 

5 If sufficient computational power and storage capacity are available in the wireless 

device 100, further processing of the metadata vector 138 can take place in the context 
inference engine 136, toward the objective of producing the result of an inferred current 
context. However, if at some point in the computation, the context inference engine 136 
needs the processing power or storage capacity available at the network server 140, the 

10 metadata vector 138 is sent from the wireless device 100 to the context inference engine 
142 in the network server 140 of Figure 2A. The context inference engine 142 in the 
network server 140 can perform the required processing on the metadata vector 138 and 
then return it to the context inference engine 136 in the wireless device 100 for 
completion of the an inferred current context result. Alternately, the context inference 

15 engine 142 in the network server 140 can complete the required processing and then 
return the resultant inferred current context to the wireless device 100. 

Figure 2 shows the architecture of a wireless device with support for context 
awareness. The context awareness is built on top of sensory information received from 

20 various types of sensors physically located in the handset shown in Figure 1 . The sensors 
shown include POSITIONING SENSOR 122, TOUCH SENSOR 124, AUDIO SENSOR 
125, COMPASS SENSOR 126, AMBIENT LIGHT SENSOR 128, AMBIENT 
TEMPERATURE SENSOR 132, and THREE- AXIS ACCELERATION SENSOR 134. 
The sensors can also be located in accessory-like phone covers or in a wireless accessory 

25 such as a Bluetooth enabled device. The sensors may also be located in the environment 
such as in the user's rooms or vehicles. Also the time duration of use of a phone and other 
available information can be used along with sensor data in context awareness services. 

Figure 2 shows sensor data received from the sensors 122, 124, 125, 126, 128, 
30 132, and 134 is processed by Context Inference Engine 136. The application programs A, 
B, X, or Y running in the wireless device 100, may optionally provide application data to 
the context inference engine 136, along with their request for current context. The context 
inference engine 136 can optionally process the sensor signals and the application data to 
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produce the current context. The context inference engine 136 then feeds the current 
context through various APIs 154, 156, 158, and 162 to application programs A, B, X, 
and Y. The application programs may register themselves at the Application 
Programming Interface 154 to receive current context or changes in the context. This 
5 enables context sensitivity in the application programs. 

Figure 2 shows "native" application programs A and B which are executed in a 
first software system SSI of the wireless device 100. The term "Software System" is used 
here for any environment with execution capability. This first software system may be 
10 proprietary or based on a commercially available real-time operating system, such as 

NOS, ISA, EPOC, JAVA, or WAP. Third party application programs X and are executed 
within an execution environment. This execution environment may limit the system 
capabilities available for the application programs, such as access to APIs (fixed, not 
dynamic behavior). 

15 

Figure 2 shows the mobile user's privacy control feature. The privacy control 
feature enables the user to designate which application programs are granted access to the 
context awareness APIs 154 to utilize the current context information produced by the 
context inference engine 136. All requests or registrations by application programs A, B, 

20 X, and Y to have access to the Context Inference Engine 136, must first go through the 
Privacy Control block 150. Privacy Control block 150 uses the user's security data check 
stored in the Privacy Profile 152 to grant access rights to the requesting application 
programs. The user controls the granting of access rights by means of the user's security 
data input by the user through the user interface. The user's security data includes 

25 permissions list 155, Public Key Infrastructure (PKI) certificates 157, PKI trusted 

authority trust list 159, and flags set by the user for those application programs that have 
been authenticated by the PKI procedures, data set 161. The user can update the user's 
security data with the UPDATE PRIVACY FEATURES menu displayed by the wireless 
device 100 shown in Figures 1A and IB. Access might be granted to an application 

30 program based on its digital signature, which is a part of the system applications, or other 
means known in the art. It is also possible to provide a separate system-wide Privacy User 
Interface to the privacy control 150, which can be employed by the mobile user to set the 
privacy policies and to alert the mobile user that an application program is attempting to 
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register to receive the user's private context awareness information. The privacy control 
150 and Privacy Profile 152 enable the mobile user to grant, deny, or revoke access, to 
grant access for a limited time, or to require an application program to always request 
registration before the user grants access. 

5 

In Figure 2, the Context Inference Engine 136 in the wireless device 100 makes 
inferences from all the sensor inputs based on where the wireless device is located by the 
mobile user. For instance the inferred current context of the device 100 may be "IN THE 
USER'S POCKET", when a certain set of sensors input a specific combination of signals 
10 having a specific value range. As an example, the resulting inference of the current 
context by the Context Interference Engine 136 could be expressed in XML language 
format as follows: 

<Context Inference Engine in Device> 

<device placement pocket <l device placement 
15 <User Interface state> sleep mode </User Interface state> 

< device location> in elevator 5 building 1 floor 2<l device location> 

<API active actions> meeting starting on floor 3 room 322 </API active actions> 
</Context Inference Engine in Device > 

The Context Inference Engine 136 in the wireless device 100 can perform the 
20 context inference process with any of several methods. Different input information from 
the sensors can be weighted according to their relative value of importance appropriate for 
each environment condition or situation to be analyzed. Each sensor has it's own weight 
value. Alternatively, the weight values for each sensor for each environment condition 
can be learned from training sessions using, for example artificial neural networks 
25 (ANNs), self-organizing maps (SOMs), decision trees, fuzzy rule-based systems, or 

model-based systems such as Hidden Markov Modeling (HMM). Combinations of two or 
more of the alternate methods can be used, depending on the application. 

The Context Inference Engine 136 can continuously adapt its weights through 
30 adaptive and continuous learning methods, where the user teaches the wireless device 100 
new environment conditions and names them. Hidden Markov Modeling (HMM) can be 
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used, for example, to implement an adaptive and continuous learning method for the 
Context Inference Engine 136. Alternately, the wireless device 100 can be programmed 
to spontaneously recognize a changed scene by comparing it with known scenes. The user 
can teach the wireless device new environmental conditions and name them, using the 
5 adaptive and automatic learning capability of neural networks. Adaptive and continuous 
learning methods are computationally intensive and are appropriate candidates to place on 
the network server 140, which assists the wireless device 100, as discussed below. 

The field of context inference has applied the principles of automated pattern 
10 recognition to processing diverse types sensor inputs. Speech recognition has been 
applied to processing speech signals and handwriting recognition has been applied to 
processing hand force and accelerometer signals. In the field of robotics, image 
recognition has been applied to processing digitized still and motion images, mechanical 
location recognition has been applied to processing laser and sonar range finder signals, 
15 and mechanical motion recognition to has been applied to processing inertial, 

acceleration, and heading signals. In the field of prosthetic devices, touch recognition has 
been applied to processing tactile sensor signals. In the field of medicine, automated 
diagnostic programs recognize various pathologies by processing bioelectric field signals, 
as well as the more traditional pulse, respiration rate, and body temperature signals. 
20 These diverse sensor signal recognition processes have the common feature that an initial 
training stage is conducted where sampled signals are equated with a statistical model for 
those signals. 

The principles of automated pattern recognition for these diverse sensor inputs are 
25 exemplified by the techniques for recognizing speech patterns. A common technique 
used in speech recognition is Hidden Markov Modeling (HMM). The term "Hidden" 
refers to the probabilistic and not directly observable events which underlie a speech 
signal. HMM speech recognition systems typically use realizations of phonemes which 
are statistical models of phonetic segments having parameters that are estimated from a 
30 set of training examples. Models of words are made by chaining or linking appropriate 
statistical models of phonetic segments. The statistical models serve as standards which 
are to be matched with the unknown voice signals to be recognized. 
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Recognition of unknown voice signals requires sampling and digitizing the 
speaker's spoken phonemes. These digitized phonemes are then processed into metadata. 
The metadata is then compared with the standard statistical models of phonemes. The 
most likely matches are then the inferred speech recognition result. 

5 

Recognition consists of finding the most likely path through the set of word 
models for the input speech signal. HMM speech recognition decoding systems first need 
to be trained through an iterative process. The system must be exposed to training 
examples or words of a particular speaker's voice. A training word is analyzed to generate 
10 a framed sequence of acoustic parameters or statistical models. A valid or "good" 
recognition occurs when the most likely path through the set of word models for the 
training word results in recognizing the correct training word. 

Some useful references discussing the principles of Hidden Markov Models are: 
15 Rabiner, L. R., "A tutorial on hidden Markov models and selected applications in 

speech recognition", Proceedings of the IEEE , volume 77, number 2, 1989, pages 257- 
286. 

Rabiner, L. R. and Juang, B. H., "An introduction to hidden Markov models", 
IEEE ASSP Magazine , January 1986, pages 4-15. 
20 Fraser, Andrew M. and Dimitriadis, Alexis, "Forecasting Probability Densities by 

Using Hidden Markov Models with Mixed States", Time Series Prediction: Forecasting 
the Future and Understanding the Past , Addison-Wesley, editor Weigend, Andreas S. and 
Gershenfeld, Neil A., 1994. 

Charniak, Eugene, Statistical Language Learning , MIT Press, Cambridge, 
25 Massachusetts, 1993. 

To illustrate how Hidden Markov Modeling (HMM) can be extended beyond 
speech recognition, an example is given here for touch recognition. In the training stage 
for touch recognition, tactile sensor signals are input from touching a tactile transducer to 
30 a rough texture, such as for example sandpaper. The tactile sensor signals are 

transformed into a statistical model of the input signal. The statistical model is stored as a 
standard in a computer memory under the handle "rough_texture". To expand the range 
of sensor signals that are included in the model for "rough_texture", several training 
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sessions can be conducted, each with a different direction or pressure for touching the 
sandpaper, resulting in several different samples of the statistical model. The set of 
samples of the statistical model are stored as a standard under the handle "rough_texture". 
Other training sessions are conducted with a smooth texture, such as glass. The tactile 
5 sensor signals input from touching the tactile transducer to the smooth texture are 

transformed into a statistical model of the input signal and stored as a standard under the 
handle "smooth_texture". Later, in the recognition mode, an unknown object is touched 
by the tactile transducer resulting in a sample tactile sensor signal. Recognition of 
unknown touch signals requires sampling and digitizing the touch transducer's signals. 
10 These digitized sensor signals are then processed into metadata. The metadata is then 
compared with the standard statistical models of "rough_texture" and "smooth_texture". 
The most likely match is then the inferred touch recognition result. 

Combinations of two or more types of sensors can have their signals combined 
into an input metadata vector that characterizes a composite sampling event. The 
composite sampling event can be recognized using the principles of Hidden Markov 
Modeling (HMM). An example composite sampling event can be the state of the health 
and fatigue of the user of a wireless device 100. For example, a wireless device 100 can 
be equipped with a tactile transducer which outputs tactile sensor signals in response to 
the hand force and pulse rate of the user who is gripping the wireless device 100. The 
wireless device 100 can be equipped with a temperature sensor which outputs body 
temperature signals in response to the user gripping the wireless device 100. Hidden 
Markov Modeling (HMM) can be used to recognize a force/temperature input metadata 
vector that characterizes the combination of the hand force and the temperature sensor 
signals resulting from a sampling event. A composite sampling event in this example can 
have an extended duration so that the force sensor can transduce the pulse rate of the user 
over a period of time. 

In the training stage, the tactile sensor signals and the force sensor signals are 
30 output while the user is in a condition of good health and resting normally. The tactile 
sensor signals and the force sensor signals are combined into a force/temperature input 
metadata vector which is transformed into a statistical model of the input signals. The 
statistical model is stored as a standard in the computer memory of the wireless device 
18 
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100 under the handle "good_health_ resting_normally". Other training sessions are 
conducted with the user in different states of health and fatigue. For example, the user 
may be training the wireless device 100 while working late at night at the office. The 
tactile sensor signals and the force sensor signals resulting from holding the wireless 
5 device 100, are combined into a force/temperature input metadata vector for the user in 
the condition of being in good health but fatigued. The force/temperature input metadata 
vector is transformed into a statistical model of the input signals and stored as a standard 
under the handle "good_health_ fatigued". 

10 Later, in the recognition mode, as the user holds the wireless device 100, the 

tactile sensor signals and the force sensor signals are sampled. The Health/Fatigue_State 
recognition consists of sampling and digitizing the touch transducer's signals. These 
digitized sensor signals are then processed into a metadata vector. The metadata vector is 
then compared with the standard statistical models of handle "good_health_ 

15 resting_normally" and "good_health_ fatigued". The most likely match is then the 
inferred touch recognition result. 

In accordance with the invention, this recognition result can be used by a health 
maintenance application program in the wireless device 100, to provide useful and 
20 appropriate information to the user. For example, a health maintenance program can 
process the recognition result, and in response, signal an alarm to the user and provide 
suggestions for medications to palliate the sensed fatigue. One problem with automatic 
recognition programs is that they are either relatively large or they call databases that are 
relatively large in comparison to the memory capacity of the wireless device 100. 

25 

Another aspect of the invention is the recognition result can be used by a 
supplementary application program in a remote server, to provide additional and more 
detailed useful and appropriate information to the user. For example, the server can 
access a large database of suggestions for medications to palliate the sensed fatigue of the 
30 user. The results of the search of the database can be returned to the wireless device 100. 
The server can also maintain a personal profile of the user's characteristics and 
preferences and it can use that profile in automatically formulate its query to the database. 
For example, the user's drug allergies can be stored in the server's database, to insure that 
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recommendations are not made that will result in an allergic reaction by the user to the 
suggested medication. 

Figure 2A is a functional block diagram of the wireless device 100, the server 140, 
5 and the web server 160, and their interaction when exchanging the metadata vector 138 
and the privacy control data 150'. These exchanges are bulk encrypted with a symmetric 
session key, such as a Data Encryption Standard (DES) key, to protect the privacy of the 
data. To insure the integrity of the metadata vector 138 and the privacy control data 150', 
a message authentication code (MAC) can be computed and appended to the data, as 
10 described in the above referenced book by Stephen Thomas entitled "SSL and TLS", 
published by John Wiley and Sons, 2000. To insure that the source of the metadata 
vector 138 and the privacy control data 150' cannot be repudiated, a digital signature can 
be appended to the data, as described in the above referenced book by Richard E. Smith 
entitled "Internet Cryptography", published by Addison Wesley, 1997. 

15 

Figure 2A shows the scope of the distributed context awareness implementation. 
The wireless device 100 has context sensitive applications A, B, X, and Y either 
downloaded or in firmware. The wireless device 100 may locally preprocess part of the 
context information in the metadata vector 138 before sending it to the context inference 
20 engine 142 in the network server 140 which is capable of processing the data and 

responding back with the resulting current context. The wireless device 100 may run 
application programs that require accessing the web service server 160 to provide context 
sensitive services to the mobile user. 

25 Figure 2A shows how processing of sensor data from the sensors in the wireless 

device 100, can be distributed between the wireless device and the network server 140. 
The operation in Figure 2A is as follows: 

1. The sensors continuously provide the sensor data to the Context Inference 
30 Engine 136 in the wireless device 100. 

2. An application program that utilizes the context awareness APIs 154 may 
request the latest context information, or the application program may be registered to 
receive any changes to specific context information. 
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3. The Context Inference Engine 136 securely contacts the Context Inference 
Engine 142 of the network server 140 and sends the metadata vector 138 to the server 
140. Depending on the sensors and the implementation details, Context Inference Engine 
136 may preprocess part of the sensor data in the metadata vector 138 prior to sending it. 

5 Depending on the sensors and the interval for processing, there may be virtual connection 
open between Context Inference Engine 136 and Context Inference Engine 142 for 
frequent data exchanges. Context Inference Engine 142 at the network server 140, has the 
processing power and memory capacity to handle computationally intensive and/or 
memory intensive processing of the preprocessed sensor data in the metadata vector 138 
10 to produce the current context result information. 

4. Context Inference Engine 142 at the network server 140 may utilize local 
user information (history information, customer details) stored in the user database 146 
for making a more accurate determination of the mobile user's current context. 

5. Context Inference Engine 142 at the network server 140 then securely 

15 returns the current context awareness information to Context Inference Engine 136 in the 
wireless device 100. 

6. Context Inference Engine 136 in the wireless device 100 then provides the 
current context awareness information through Context Awareness APIs 154 to the 
application programs registered for to receive that information. 

20 

Figure 2A shows how Web Services in Web Service Server 160 are enabled to 
receive current context results of the wireless device 100. Web Services Server 160 has a 
software system for server application program A and an execution environment for 
server application programs X and Y that are similar to the software system SSI and 
25 execution environment (Exec. Env.) in the wireless device 100 shown in Figure 2. Server 
Application programs A, X, and Y in Web Service Server 160 may require access through 
the Context Awareness APIs to provide Web Services with the current context of the 
wireless device 100. 

30 In Figure 2A, Web Service Server 160 uses the Context Inference Client 176 to 

contact the Context Inference Server 174 in the network server 140. Context Inference 
Client 176 may utilize customer database information in database 184 to enhance the 
context sensitivity capabilities of the web server 160. The contact to the network server 
21 
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140 is done through a context awareness interface 186 to the Context Inference Server 
174 in the network server 140. 

Context Inference Server 174 registers the Web Services of the web server 160 
5 through the privacy control 164 of the network server 140 to the Context Inference Engine 
142. Privacy control 164 has a cached copy 144 of the Privacy Profile 152 of the wireless 
device 100. This enables processing of the privacy check in the network Server 140 for 
access requests from web server 160. The communication between web server 160 and 
network server 140 is secured using the Internet secure protocols such as HTTPS or SSL. 
10 The Context Inference Server 174 can publish its own service as a Web Service to other 
Web Services on the Internet, in which case the implementation of the interface 186 
between web server 160 and network server 140 can be Extensible Markup Language 
(XML) messages carried in the Simple Object Access Protocol (SOAP) messaging 
protocol. 

15 

The Context inference Engine 142 in the network server 140 will receive 
processed sensor metadata vector 138 information and possibly some application API 
information originated from the Context Inference Engine 136 of the wireless device 100. 
The Context inference Engine 142 of the network server has user database 146 

20 information of the behavior of the user and of the past usage of the wireless device. The 
Context inference Engine 142 of the network server may also have third party services 
available (such as instances offering content and/or services) to be offered to potential 
users. What is offered to the user can also depend on the user profile 144. The nature of 
the Context inference Engine 136 information of the wireless device 100 that is conveyed 

25 to the Context inference Engine 142 of the network can be controlled with the privacy 
control 150 that is managed by the user of the wireless device 100. The user may thus 
fully or partly disable the Context inference Engine 142 of the network to control the 
amount of his/her information that can be used by third party services. The privacy 
control 150 enables the user to control access by anyone to his/her private information. 

30 The Context inference Engine 136 of the wireless device receives an input from 

the API interface 154 from the applications A, B, X, or Y located in the wireless device 
100. An example would be from a calendar application program indicating that a meeting 
is starting in 25 minutes time. As another example the calendar application program 
22 
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indicates that Lisa is having a birthday tomorrow into which you are participating. The 
Context inference Engine 136 of the wireless device can convey processed result 
information to the Context inference Engine 142 of the network server. Now in addition 
to the sensor information, information from the application programs A, B, X, or Y can 
5 also be used in the decision making of the Context inference Engine 136 of the wireless 
device. A combination of the sensor information and information coming from the 
application programs A, B, X, or Y can be processed by the Context inference Engine 
136. The user's behavior or usage patterns can be detected from the sensor and recorded in 
a the user database, concerning the usage of the application programs. As previously 

10 discussed, the processing of this combined information from the sensors and from the 
application programs can be shared between the Context inference Engine 136 and the 
Context inference Engine 142. Either the application programs A, B, X, or Y running in 
the wireless device 100 or the server application programs A, X and Y running in the web 
server 160, may optionally provide application data to the context inference engine 142 in 

15 the network server 140. The context inference engine 142 can optionally process the 
metadata vector 138 and the application data to produce the current context. 

The information transfer from the Context inference Engine 136 of the wireless 
device to the Context inference Engine 142 of the network server can be done in 

20 alternative ways. The system can be managed so that the current consumption and transfer 
capacity between the wireless device 100 and the network server 140 is taken into 
account. The context information does not always have to be collected so frequently that 
it would have to be periodically transferred to the network side 140 every few seconds. 
Depending on the application, the timing window applied to information transfer from the 

25 Context inference Engine 136 of the wireless device 100 to the Context inference Engine 
142 of the server 140 can vary from seconds to minutes. If there were no event change or 
condition change in the environment of the wireless device 100, there would be no need 
to transfer information to the Context inference Engine 142 of the server 140. 
Additionally information can be temporarily stored in a buffer in the wireless device 100, 

30 which can then transferred less frequently to the network Context inference Engine 142. 
Packet based GPRS and UMTS can support the less frequent information transfer rates. 
Also, it is advantageous to send the network Context inference Engine 142 information 



23 



WO 02/093877 PCT/IB02/01550 

from the wireless device 100 as an attachment, immediately subsequent to other signaling 
made to in the network direction from the wireless device 100, thus saving the radio 
transmitter of the wireless device 100 from having to be switched on again for transferring 
the Context inference Engine 136 information separately to the network server 140. 

5 

Returning to Figure 1, the relationship is shown between the network server 140, 
the Universal Description, Discovery and Integration (UDDI) registry 170, and a plurality 
of web site servers 160. UDDI is a defacto standard for an Internet-based registry. The 
UDDI registry 170 enables the network server 140 to discover new web sites for services 
10 and businesses on the Internet. Once such services and businesses are identified by the 
UDDI registry 170 to the network server 140, then the server 140 must apply the mobile 
user's cached privacy profile 144 in Figure 2A, in order to prevent unauthorized access of 
the user's private data by application programs on the newly discovered web sites. 

15 Figure 3 is a network process flow diagram of the interaction of the wireless 

device 100 1 the first column, network server 140 in the middle column, and web server 
160 in the right column, when they carry out the determination of the current context of 
the wireless device 100. The process begins with the wireless device 100 in step 302: 

20 Step 302: PRIVACY CONTROL 150 IN WIRELESS DEVICE 100 SENDS 

UPDATED PRIVACY PROFILE TO NETWORK SERVER 140. 

Then the network server 140 continues with step 304: 

25 Step 304: NETWORK SERVER 140 UPDATES CACHED PRD/ACY PROFILE 

144. 

The wireless device 100 continues with the following steps 306, 308, and 310: 

30 Step 306: SENSORS CONTINUOUSLY PROVIDE SENSOR DATA TO 

CONTEXT INFERENCE ENGINE 136 IN WIRELESS DEVICE 100. 

Step 308: APPLICATION PROGRAM THAT USES CONTEXT AWARENESS 
API 154 REQUESTS LATEST CONTEXT INFORMATION. 

35 

Step 310: CONTEXT INFERENCE ENGINE 136 CONTACTS CONTEXT 
INFERENCE ENGINE 142 OF THE NETWORK SERVER 140 AND SENDS THE 
METADATA VECTOR 138 TO SERVER 140. 
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Then the network server 140 continues with steps 312 and 314: 



Step 312: CONTEXT INFERENCE ENGINE 142 AT NETWORK SERVER 140 
USES LOCAL USER INFORMATION STORED IN USER DATABASE 146 TO 
5 MAKE A MORE ACCURATE DETERMINATION OF THE MOBILE USER'S 
CURRENT CONTEXT. 



Step 314: NETWORK SERVER 140 REQUESTS DATA FROM WEB SERVER 

160. 

10 THE NETWORK SERVER'S ACCESS IS AUTHORIZED BY CACHED 

PRIVACY PROFILE 144 IN NETWORK SERVER. 



Then the web server 160 continues with step 316: 



15 Step 316: WEB SERVER PROVIDES USER INFORMATION STORED IN 

DATABASE 184 TO NETWORK SERVER 140. 

Then the network server 140 continues with step 318: 

20 Step 318: CONTEXT INFERENCE ENGINE 142 AT THE NETWORK 

SERVER 140 THEN SECURELY RETURNS THE CURRENT CONTEXT 
AWARENESS INFORMATION TO CONTEXT INFERENCE ENGINE 136 IN THE 
WIRELESS DEVICE 100. 

25 Then the wireless device 100 finishes with step 320: 

Step 318: CONTEXT INFERENCE ENGINE 136 IN THE WIRELESS DEVICE 
100 THEN PROVIDES THE CURRENT CONTEXT AWARENESS INFORMATION 
THROUGH CONTEXT AWARENESS APIs 154 TO THE APPLICATION 
30 PROGRAMS REGISTERED TO RECEIVE THAT INFORMATION. 

Figure 4 is a functional block diagram of the network server 140, showing the 
memory 402 storing the application services software programs needed to perform the 
operations of the invention. The memory is connected by the bus 404 to the cache 144, 
35 user database 146, TCP/TP network adapter 406, and central processor 410. The memory 
402 stores programs which are sequences of executable instructions which, when 
executed by the processor 410, carry out the methods of the invention. 

Figure 4 is a functional block diagram of the network server, showing the memory 
40 storing the application services software programs needed to perform the operations of an 
embodiment of the invention. Figure 4 discloses the functional components of an 
exemplary network server 140 arranged as an object model. The object model groups the 
25 
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object oriented software programs into components that perform the major functions and 
applications in network server 140. The object model for memory 402 of network server 
140 employs a three-tier architecture that includes presentation tier 415, infrastructure 
objects partition 422, and business logic tier 414. The object model further divides 
5 business logic tier 414 into two partitions, application objects partition 422 and data 
objects partition 426. 

Presentation tier 415 retains the programs that manage the device interfaces to 
network server 140. In Figure 4, presentation tier 415 includes network interface 420. A 

10 suitable implementation of presentation tier 415 may use Java servlets to interact with 
WAP protocol gateway 120 via the hypertext transfer protocol ("HTTP"). The Java 
servlets ran within a request/response server that manages the exchange of messages 
between WAP protocol gateway 120 and network server 140. A Java servlet is a Java 
program that runs within a Web server environment. A Java servlet takes a request as 

15 input, parses the data, performs logic operations, and issues a response back to WAP 
protocol gateway 120. The Java runtime platform pools the Java servlets to 
simultaneously service many requests. Network interface 420 accepts request messages 
from WAP protocol gateway 120 and passes the information in the request to visit object 
428 for further processing. Visit object 428 passes the result of that processing to network 

20 interface 420 for transmission back to the WAP protocol gateway 120. Network interface 
420 may also use network adapter 406 to exchange data with another user device. 

Infrastructure objects partition 422 retains the programs that perform 
administrative and system functions on behalf of business logic tier 414. Infrastructure 
25 objects partition 422 includes operating system 425, and an object oriented software 

program component for database server interface 430, and system administrator interface 
432. 

Business logic tier 414 in Figure 4 includes multiple instances of visit object 428, 
30 428', 428". A separate instance of visit object 428 exists for each network interface 420 
session. Each visit object 428 is a stateful session object that includes a persistent storage 
area from initiation through termination of the session, not just during a single interaction 
or method call. The persistent storage area retains information associated with the session. 

26 
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When WAP protocol gateway 120 sends a metadata vector 138 message to 
network server 140, the message is sent to network interface 420 to invoke a method that 
creates visit object 428 and stores connection information as a state in visit object 428. 
5 Visit object 428 may, in turn, invoke a method in context inference engine 142 

application 440 to perform a context inference on the metadata vector and return a current 
context result. 

When WAP protocol gateway 120 sends a privacy control data 150' message to 
10 network server 140, the message is sent to network interface 420 to invoke a method that 
creates visit object 428 and stores connection information as a state in visit object 428. 
Visit object 428 may, in turn, invoke a method in privacy control 164 application 442 to 
update the cached privacy profile 144. The application 442, in turn make a method call to 
privacy profile update application 448 to store the updated data 150' in the cache 144. 

15 

When WAP protocol gateway 120 sends a user data update message to network 
server 140, the message is sent to network interface 420 to invoke a method that creates 
visit object 428 and stores connection information as a state in visit object 428. Visit 
object 428 may, in turn, invoke a method in user database application 446 to store the user 
20 data in the database 146. 

A description of server programming applications developed with Enterprise Java 
Beans is provided in the book by Ed Roman entitled "Mastering Enterprise Java Beans", 
published by John Wiley and Sons, 1999. A description of the use of an object model in 
25 the design of server applications is provided in the book by Matthew Reynolds entitled 
"Beginning E-Commerce", Wrox Press Inc, 2000, (ISBN: 1861003986). Java servlets and 
the development of web site servers is described in the book by Duane K. Fields, et al. 
entitled "Web Development with Java Server Pages", published by Manning Publications 
Co., 2000. 

30 

The resulting context sensitive web services invention enables a mobile phone or 
wireless device 100 to use context inference techniques to sense the user's environment 
and in response, to provide useful information to the user that is appropriate to the user's 
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perceived environment. The mobile user is provided with the ability to control access by 
application programs anywhere in the network, to the user's private data. 

Although a specific embodiment of the invention has been disclosed, it 
5 will be understood by those having skill in the art that changes can be made to the specific 
embodiment without departing from the spirit and the scope of the invention. 
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CLAIMS 

What is claimed is: 

5 1 . A method to enable a wireless device to provide useful information to its 

user that is appropriate to the device's current environment, comprising: 

receiving sensor signals characterizing a current environment of the wireless 

device; 

processing the sensor signals with a context inference engine; 
10 outputting a current context result from the processing by context inference 

engine; and 

providing useful information to the user in response to the current context result. 

2. The method of claim 1, wherein the processing of the sensor signals with a 
15 context inference engine is embodied as programmed instructions executed within the 

user's wireless device. 

3. The method of claim 1, wherein the processing of the sensor signals with a 
context inference engine is embodied as programmed instructions executed within a 

20 separate network server in response to signals from the user's wireless device. 

4. The method of claim 3, wherein the server accesses files from a web 
server, for selective forwarding to the user's wireless device. 

25 5. The method of claim 3, wherein the wireless device offloads a portion of 

the processing of the sensor signals with a context inference engine to the server. 

6. The method of claim 3, wherein a personal profile of the user is maintained 
by the server. 

30 

7. The method of claim 1, which further comprises: 

providing user control of access by application programs to the user's private data. 
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8. The method of claim 3, which further comprises: 

providing user control of access by application programs to the user's private data 
in the server. 

9. The method of claim 1, which further comprises: 

providing user control of access by application programs in a web server, to the 
user's private data. 

10. The method of claim 1, which further comprises: 
providing the current context result to an application program; and 
receiving the useful information for the user from the application program. 

11. The method of claim 1, which further comprises: 

providing user control of access by an application program to the current context 

result; 

providing the current context result to the application program in response to the 
user control; and 

receiving the useful information from the application program. 

12. The method of claim 11, which further comprises: 

granting access permission to the application program to access the current context 
result, based on the user's data stored in a privacy profile. 

13. The method of claim 11, wherein providing user control of access is 
embodied as programmed instructions executed within a separate network server in 
response to signals from the user's wireless device. 

14. The method of claim 13, wherein the server accesses files from a web 
server, for selective forwarding to the user's wireless device. 
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15. The method of claim 13, wherein the wireless device offloads a portion of 
the processing of providing user control of access, to the server. 

16. The method of claim 13, wherein a personal profile of the user is 
maintained by the server. 

17. The method of claim 16, which further comprises: 

providing user control of access by application programs to the user's personal 

profile. 

18. The method of claim 13, which further comprises: 

providing user control of access by application programs to the user's personal 
profile in the server. 

19. The method of claim 11, which further comprises: 

providing user control of access by application programs in a web server, to the 
user's private data. 

20. The method of claim 19, which further comprises: 

enabling context sensitive applications and services within the wireless device 
while providing to the user control over the privacy user's current context result. 

21. The method of claim 19, which further comprises: 

enabling context sensitive applications and services within the network server 
while providing to the user control over the privacy user's current context result. 

22. An apparatus to enable a wireless device to provide useful information to 
its user that is appropriate to the device's current environment, comprising: 

a processor; 

a memory coupled to the processor, programmed to perform the steps of: 
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receiving sensor signals characterizing a current environment of the wireless 

device; 

processing the sensor signals with a context inference engine; 
outputting a current context result from the processing by context inference 
5 engine; and 

providing useful information to the user in response to the current context result. 

23. The apparatus of claim 22, wherein the processing of the sensor signals 
with a context inference engine is embodied as programmed instructions executed within 

10 the user's wireless device. 

24. The apparatus of claim 22, wherein the processing of the sensor signals 
with a context inference engine is embodied as programmed instructions executed within 
a separate network server in response to signals from the user's wireless device. 

15 

25. The apparatus of claim 24, wherein the server accesses files from a web 
server, for selective forwarding to the user's wireless device. 

26. A wireless device to provide useful information to its user that is 
20 appropriate to the device's current environment, comprising: 

a sensor for providing sensor signals characterizing a current environment of the 
wireless device; 

a context inference engine coupled to the sensor, for processing the sensor signals; 
said context inference engine providing a current context result from the 
25 processing; and 

an output device coupled to the context inference engine, for providing useful 
information to the user in response to the current context result. 

27. A wireless device to provide useful information to its user that is 
30 appropriate to the device's current environment, comprising: 

a privacy control for providing the user control of access by an application 
program to the user's private data; 

32 



WO 02/093877 



PCT/IB02/01550 



a sensor for providing sensor signals characterizing a current environment of the 
wireless device; 

a context inference engine coupled to the sensor, for processing the sensor signals; 
said context inference engine coupled to the privacy control, for providing a 
5 current context result from the processing to the application program; and 

an output device coupled to the privacy control, for providing useful information 
to the user in response to the application program. 

28. A system to provide useful information to the user of a wireless device that 
10 is appropriate to the device's current environment, comprising: 

a privacy control in a server for receiving a user privacy profile from the wireless 
device and providing the user control of access by an application program to the user's 
private data; 

a sensor in the wireless device for providing sensor signals characterizing a current 
15 environment of the wireless device; 

a context inference engine in the server coupled to the wireless device, for 
processing sensor information derived from the sensor signals; 

said context inference engine coupled to the privacy control, for providing a 
current context result from the processing to the application program; and 
20 an output device in the server, coupled to the privacy control, for transmitting 

useful information to the wireless device in response to the application program. 



29. A method to enable a wireless device to provide useful information to its 
25 user that is appropriate to the device's current environment, comprising: 

running a program that provides application data to the wireless device; 
receiving sensor signals characterizing a current environment of the wireless 

device; 

processing the sensor signals and the application data with a context inference 

30 engine; 

outputting a current context result from the processing by context inference 
engine; and 
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providing useful information to the user in response to the current context result. 

30. The method of claim 29, wherein the processing of the sensor signals with 
a context inference engine is embodied as programmed instructions executed within the 

5 user's wireless device. 

3 1 . The method of claim 30, wherein the step of running a program occurs in 
the wireless device. 

10 32. The method of claim 29, wherein the processing of the sensor signals with 

a context inference engine is embodied as programmed instructions executed within a 
separate network server in response to signals from the user's wireless device. 

33. The method of claim 32, wherein the step of running a program occurs in a 
15 web server coupled to the network server. 

34. A system to enable a wireless device to provide useful information to its 
user that is appropriate to the device's current environment, comprising: 

a processor; 

20 a memory coupled to the processor, programmed to perform the steps of: 

running a program that provides application data to the wireless device; 
receiving sensor signals characterizing a current environment of the wireless 

device; 

processing the sensor signals and the application data with a context inference 

25 engine; 

outputting a current context result from the processing by context inference 
engine; and 

providing useful information to the user in response to the current context result. 

30 35. The system of claim 34, wherein the processing of the sensor signals and 

the application data with a context inference engine is embodied as programmed 
instructions executed within the user's wireless device. 
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36. The system of claim 35, wherein the step of running a program occurs in 
the wireless device. 

5 

37. The system of claim 34, wherein the processing of the sensor signals and 
the application data with a context inference engine is embodied as programmed 
instructions executed within a separate network server in response to signals from the 
user's wireless device. 

10 

38. The system of claim 37, wherein the step of running a program occurs in a 
web server coupled to the network server. 

39. A method to enable a wireless device to provide useful information to its 
15 user that is appropriate to the device's current environment, comprising: 

receiving sensor signals characterizing a current environment of the wireless 

device; 

partially processing the sensor signals with a first context inference engine within 
the user's wireless device; 
20 sending the partially processed sensor signals to a second context inference engine 

within a separate network server; 

completing the processing the sensor signals with the second context inference 

engine; 

sending a current context result from the second context inference engine to the 
25 wireless device; and 

providing useful information to the user in response to the current context result. 

40. A system to enable a wireless device to provide useful information to its 
user that is appropriate to the device's current environment, comprising: 

30 a wireless device for receiving sensor signals characterizing a current environment 

of the wireless device; 
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a first context inference engine within the user's wireless device for partially 
processing the sensor signals; 

a second context inference engine within a separate network server for receiving 
the partially processed sensor signals sent from the wireless device and completing the 
processing the sensor signals; 

said second context inference engine sending a current context result from the 
second context inference engine to the wireless device; and 

said wireless device providing useful information to the user in response to the 
current context result. 

41 . A method to enable a wireless device to provide useful information to its 
user that is appropriate to the device's current environment, comprising: 

receiving sensor signals characterizing a current environment of the wireless 

device; 

partially processing the sensor signals with a first context inference engine within 
the user's wireless device; 

sending the partially processed sensor signals to a second context inference engine 
within a separate network server; 

completing the processing the sensor signals with the second context inference 

engine; 

forwarding a current context result from the network server to a second server; and 
sending useful information from the second server to the user's wireless device in 
response to the current context result. 
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FIG. 4 
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